Repository navigation
Conversation
| INSERT INTO dv VALUES (1, 'c'), (4, 'w'); | ||
| -- snapshot 4: (1, c) (2, b) (3, x) (4, w) (5, z) | ||
| DELETE FROM dv WHERE id IN (2, 5); | ||
| -- snapshot 6: (1, c) (3, x) (4, w) |
There was a problem hiding this comment.
Thanks for adding the test data and integration tests! However, even though the test data construction removes id IN (2, 5), the actual dataset doesn’t produce any DV. This might be because the dataset is too small, and during compaction the files get merged directly without generating DV. Please adjust the test-data generation to ensure DV is covered.
Tip: You can try making the v field extremely large or increasing the number of rows to satisfy the DV conditions.
| if (!table_schema->PrimaryKeys().empty() && context->IsStreamingMode()) { | ||
| return Status::NotImplemented( | ||
| fmt::format("do not support pk table bucket={} in streaming scan process", | ||
| core_options.GetBucket())); |
There was a problem hiding this comment.
Loosening this condition seems to also enable KEY_DYNAMIC in the cross-partition mode. Please confirm whether this scenario should be supported. If yes, please add the corresponding integration test.
There was a problem hiding this comment.
Also, it seems ResolveBucketMode should distinguish between BucketMode.KEY_DYNAMIC and BucketMode.HASH_DYNAMIC based on crossPartitionUpdate.
There was a problem hiding this comment.
Could we also add an integration test for the case without DV but requiring MOR? Since the DV switch on/off seems to be orthogonal to dynamic buckets.
Purpose
Linked issue: close #493
NewDataTableScanrejected every scan of a primary-key table withbucket = -1(
do not support pk table bucket=-1 in scan process). Such tables (dynamic bucket, written byPaimon Java from Flink or Spark) can be read in batch mode: batch planning only uses the buckets
recorded in the manifest entries and does not need the dynamic bucket assigner. Since #494,
deletion vectors of these tables are combined per (partition, bucket), as Java does for
BucketMode.HASH_DYNAMIC.This change keeps rejecting dynamic bucket primary-key tables only in streaming mode, and allows
batch scans. The streaming error message now says
streaming scan process.Writing dynamic bucket primary-key tables and streaming reads stay out of scope.
Tests
ScanAndReadDynamicBucketInteTest.TestBatchReadPkDynamicBucketWithDeletionVectors: reads atable written by Spark 3.5.3 with Paimon 1.3.2 (
test/test_data/parquet/pk_dynamic_bucket_dv.db,bucket = -1,deletion-vectors.enabled = true,dynamic-bucket.target-row-num = 2, see itsREADME). Keys are spread over several buckets, each with its own deletion vector index. The test
reads three snapshots (insert, upsert, delete), checks that the plan has more than one split,
and compares the rows with what Spark returns.
ScanInteTest.TestScanPKWithInvalidOptions: streaming scans of dynamic bucket primary-keytables are still rejected, with the updated message.
Without the change in
table.cpp, the new test fails withdo not support pk table bucket=-1 in scan process.Verified on Ubuntu 24.04 (aarch64), GCC 13, Debug:
paimon-scan-inte-test(87 tests),paimon-scan-and-read-inte-test(512 tests) andpaimon-core-test(2536 tests) pass.API and Format
No public API change. No storage format or protocol change.
Documentation
No.
Generative AI tooling
Generated-by: Claude Code (Claude Opus 5.5)